fix(libkernel): return EINVAL for pthread mutex and cond operations on a destroyed object - #6244

Replaces #4971 (closed 14.10. with the note to reopen with a title ID and a log line; GitHub does not let me reopen it). Same branch, same head 9284312ac.

What

Locking, trylocking and unlocking a destroyed mutex, and signalling, broadcasting, waiting on or destroying a destroyed condition variable threw a C++ exception into the guest (Mutex has been destroyed, Condition variable has been destroyed, Condition variable has already been destroyed), which terminates the process. FreeBSD 11.4 libthr returns EINVAL in every one of these cases:

• thr_mutex.c: check_and_init_mutex returns EINVAL when *m == THR_MUTEX_DESTROYED (lock, timedlock, trylock); mutex_unlock_common returns EINVAL for it; _pthread_mutex_destroy returns EINVAL for an already destroyed mutex.
• thr_cond.c: CHECK_AND_INIT_COND returns EINVAL for THR_COND_DESTROYED (signal, broadcast, wait); _pthread_cond_destroy sets EINVAL for an already destroyed condition variable.

The change makes resolveMutex and resolveCond report the destroyed state and the callers return SCE_KERNEL_ERROR_EINVAL (the POSIX wrappers map it to EINVAL). The mutex destroy path and the condition wait path (mutexOwnership) already returned EINVAL for a destroyed mutex; this makes the rest consistent. The sce* entry points share these functions, so they change too; I did not check what a PS5 returns there, the choice follows the shared libthr behavior. This includes scePthreadCondSignalto: it now returns SCE_KERNEL_ERROR_EINVAL for a destroyed condition variable instead of throwing; for a live one it still wakes all waiters (NotifyAll) and still ignores its thread argument, as before. This follows from the code (resolveCond returns null for a destroyed condition variable); the new test does not call scePthreadCondSignalto.

Left as is: NULL pointer arguments still throw (FreeBSD would fault there), as does the use of an uninitialized condition attribute in scePthreadCondInit.

Overlap: #5429 (wake guest mutex waiters on release) is merged (main d1670009f). It rewrote the same lock call sites in Mutex.cpp; the branch now contains a merge commit of main (9284312ac) that resolves the conflict there: the three lock paths (pthread_mutex_timedlock's MutexOperations::Timedlock, scePthreadMutexLock, scePthreadMutexTimedlock) keep acquireResolved (a destroyed mutex answers EINVAL) and hand the resolved mutex to #5429's parking calls (TimedWait::ParkAcquire, TimedWait::AcquireUntil). Cond.cpp and the libkernel CMakeLists.txt merged without a conflict.

Title evidence

Ship of Harkinian (PPSA99320), reported by @milescoviello on the previous pull request #4971 (Linux, Ubuntu 24.04, GCC 13.3, Release, RTX 4070 Ti, main a077a88df + #6217): with no ROM installed the title shows its "Game files needed" dialog; pressing Cross on its OK button quits the title. On main + #6217 the quit stops with what(): Mutex has been destroyed; with the head of this branch the same quit exits with code 0. A run ended by its 75 s timeout stops the same way without the change. I have not run that title myself. The title needs #6217 (malloc_stats_fast) to get that far.

Tested

• Windows 10 (19045), MinGW-w64 GCC 15.2.0, Release, on main d92a8dad390af1807cc91622b924252141816a5a + this branch (clean merge, tree db30968fb): guest_pthread_destroyed_objects (ctest) passes with the change, ctest exit 0; without it (Cond.cpp and Mutex.cpp from main, the new test kept, libkernel rebuilt) it fails with test exit code 0xc0000409 and what(): Mutex has been destroyed, ctest exit 8.
• Earlier, on this head 9284312ac (merge of main d1670009f into the previous head 6ddb7e5c5): the 27 test executables of the targets guest_cond_*, guest_mutex_*, guest_posix_sem, guest_pthread_* and guest_semaphore all exit with code 0.
• python tools/check_conventions.py --base upstream/main passes on this head.
• Not tested: other ctest tests on main d92a8dad3, Linux, a console, the title itself. The expected values come from the FreeBSD 11.4 sources above, read from release/11.4.0 on GitHub.

Checklist

• Merges onto current main (d92a8dad3) without conflict (the branch contains a merge of main; the conflict with the merged perf(libkernel): wake guest mutex waiters on release on Windows #5429 is resolved, see Overlap)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source (n/a, none added)
• Generic behaviour, not specific to one title (the title above is the reason to reopen it, see Title evidence)
• Depends on: none
• Unblocks: PPSA99320 (Ship of Harkinian) at its quit path, once feat(libc): answer malloc_stats_fast for the default heap #6217 is merged
• AI-assisted: yes (Claude Code)

AI-assisted (Claude Code).

🤖 Generated with Claude Code